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ABSTRACT 

Crucial to curriculum development 
assessment of cognitive objective achievemert and 
of cognitive needs. Validation of instrumentation 
attainment of cognitive behavioral objectives was 
step in a state-^yide educational needs as:5i&ssmert 
the investigators. A total of 3^365 behavioral objectives 
cognitive sub jects were selected or developed by academic 
and authorities. Attainment was assessed by the state 'mandated 
testing program utilizirq nationally standart'^ized tests^ The tests 
were administered to a s .jiple of 20^000 pupils in grades four^ seven 
and eleven within Virgin*.? schools. The assessment of the test's 
content validities was accomplished by inspectional analyses which 
compared test items with selected behavioral objectives* The findings 
are described in light of the limitations of using nationally 
standardized tests to measure local performance outcomes. The 
investigators recommend the development of criterion-referenced 
exercises. (Author/SJL) 
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P u r po se_ juul_ J* r_o co_c uro_3_ 

Crucial to curriculun dcvolopnont and change is the assess- 
ment of cognitive objective achicv'onenc ind the identification 
of cognitive needs. Validation of in3tru-.on-.-ation to evaluate 
the attainment of coanitive ochavioral ob-jejl-Jves '.;as an 
essential first step in a state-v;ide educational neoas assess 
moat :jtudy conducted by the investigators. 

A total of 3,365 oehavioral objectives in 23 coqaitive 
subjects were selected or developed by acadenic specialists 
and autliorities. Because of tine constraint it was not 
possible in the first phase of the study (conpleted m 1970) 
both to develoD objectives and exercises to neasure tnen. t • 

Consequentiv, kttaim-.ient v;'as assessed by the state nandatea 
testing progran utilizing nationally standardized tests: a wiuely 
used standardized .achievement series at two levels, ^ 
standardized reading"' test, and a well-known and standardized set . 
of tests at the seconaary level. These tests together witn 2^ 
pages of charts, showing test item objective analyses are ^ 
reported in the Virgj^\j^_J.^lu^at^ 

Volum es I & I I. J.ris report nay be ordered from the State^ 
Oepartnent of Lducation, Richmond, Virginia. 

The tests wp- administered to the 20,000 sample pupils in 
grades four, 'sever, and eleven in 57 of the -state's 131 school 
divisions i.n tha six .;i3 torical-geographical regions with tne 
variables of scaool enrollment size cand population density per ^ 
square mi.le taken into account. ' " ■■ 

Project oar^meters and time constraints precluded an 
empirical validation study, such as item discrin.ina tion on pre- 
and post-training performance. Rather, the assessment of tae 
tests 's content validities was accomplished by msnectional 
analyses conparinc test items, \^;ith selected beha^oral objectives, 
•and performed by subject-matter specialists engaged uy tne 
Bureau. This orocess has sinci been computerized for expediency 
via DERTEXT, a natural language storage and retrieval system 
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tjThc criteria for solcctinq from anonq th.c iiundrcds of 
learncr-oricntoU oojoctivcs tliosc .for validatinq the stato-v/icle 
achioveraent tests were: (1) evnectation tiuit all of the sarriplc 
pupils, by grade level, would iiave haa" tiio o::)nortun 1 ty to'*>v^ 
attain sone doqrce of nastery of zr.c oojoctivos jy t.ae tine c^f^ 
the test auniinistriti a, anc: (2) no riiqhi !."icant time lause 
would iwve occurr^ Lctwocn connlotion of ru:j;iects to v/aicn tiio 
objectives pertaii a:i^ / again / ti:\o of tost auniniii tra uion . 

The following specific o;j1oc;:ivo3 , ii itcd by subject ;?,rca 
and educational level, mot the criLcria ar.,i v;orc used m the 
validation: (1) \:ork~S tudy ' and Library fi^.ills (oloix?ntary and 
secondary), (2) Reading (grades 1-11), .(J) l::u:lis:i: laruTVinqo, 
literature and aorp.pbsi tion (elOMontary , j unior nigh , hicni 
sciiool) , (4) Social Stv.uies: iii3-tory, An tnroroipgy-Sccioloqy , 
T^o^i-lticajl Science, lieoaoriies , Geography (gratles' K -4 , 5-7, 3-11), 
(5^ General i .atiic:;T;atic3 (gracics r.-4 , 4-7), anci (G) General 
Science (grades 144, 5-7). 

Finaings 

While Stake, i:bel, Co:: and Vargas, Popaan and others have 
insightfully di3,66ur3cd on norrn-'ref erenced versus criterion- 
referenced measures as to their nature and use as ncasurec in 
these days of accountability assessinent, the authors of this 
paper present findings that re\'eal how inappropriate standardized 
tests nay be for measuring taa atto^innent of cognitive 
objectives , especially v/ncn taese objectives are stated 
specifically and boiiavioral ly , av;are thoroughly of the. doui^t 
and difficulty recognized and encountered in writing objectives 
for and in assessing conple:* cognitive donains, such as 
critical tninking. * * 

Ideally in test assessment there should be 100% content 
validity waereby each jjenavioral objective is measured by a 
test item. This nay oe possible in criterion-referenced 
evaluation v/nere a specific evaluation exercise is custom- 
designed to measure a specific objective. Such could not be 
the case, in this needs assessment study using nationally ^ 
standardized achievement test items written prior t,o and 
-independent of the oujc.ctivjcs selected 6r developed, endorsed 
oy authorities and app*-.'oved for tl\e study. 

In no subject area did the tests administered to the 
sample in grades four, seven and eleven measure the objectives 
with sufficient vaJicUcy to v/arrant one-to-one assessment 
comparisons between objectives and test items. Per cent 
coverage ranged from Oj rto G7.3"s. The alternative procedure in 
the needs assessment .:tudy arready cited v;as adopted: to measure 
performance of .ti\e 'sa^;ple on the subtests and compare performance 
on these "cognitive clusters" of items with the related 
cognitive behavioral objectives. Thu^ly, cognitive needs v/ere 
identified. /' 
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'Table 1 raak orders by per cent the content: validity of 
iViO tests for neasurincj tne selected objectives. 



Table 1 

Per Cent of Objectives Measured in Terns of Subject 
Areas at Dif ferent^ educational Levels 



Reading (1-11) 




67.3% 


Social^ Studies (5-7- 




59.2% 


Social Studies (8-11) 




55.7% 


Mathematics (K-4) 




4 0.0% 


• Matheraatics (4-7) 




37.1% 


Social Studies (K-4) 




'26.9% 


V/ork-Study and Library Skills 
(Elementary) 




21.1% 


"^glish Education (High School)- 
JS'eneral Science (1-4 ) 




18.1% 
- 16.1% 


English Education (Elementary) 




13.7% 


" English Education (Junior High) 




1,3-. 7% 


General Science (5-7) 




1-3.2% 


Work-Study and Library Skils 
(Secondary) 

• 




0.0% 



None of- the test itens measured the seven^ob jectives in 
Work-Study and Library Skills at the secondary -level . At the 
Elementary level fourteen topical areas* of skill objectives 
were listed and coverage ,by the test iterjs varied from 0.0% to 
100%. Only three of chese topical skills,- though, ^were at or 
above 50 i coverage: alphabetical o'rtier, locating information 
in table of contents, and evaluation. 

The St?. ide achievement tests were most valid, in their 
measurement the cognitive objectives in Reading. Of the 52 
objectives, , . ^vere assessed by the test items or 67.3%. As 
■expected oi)jectives concerned with reading interest and 
personal development- through reading were not measured by the 
tests. * . * ^ P 
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Disappointing validation, results occurred m t^nqLou. At 
no level /ore lan«uaqo, literature and conposition objectives 
2ove?cd at or auove tac 30':-. level. For example, tao acniovcnont 
series failed to measure all of -the objectives m literature 
and only 7.6^. of^ the language objectives at the/elenentary xovei.. 
The validation results were generally similar at tae-otnor 
levels. , ' . 

•The content validity of the tests for neasurinn the Social 
Studi^^s objectives at different levels was narkodly aiviaoa: 
only 26.ro of objectives for grades K-4 were assessea, out at 
grades 5-7 and 3-11 the oojeotives were coverea respectively at 
the 59.2". and 55.7?. levels. 

Specific Mathematics objectives A/cre natched with test itens 
in the achievement series for K-4 and for 4-7.- 

The' K-4 objectives were grouped into eight skill areas vith ^ 
coverage at or above 50% ih number and numer-ation systems 
(66. 7^.) , -mputational skills (63.&%), and problem solvma • 
skills (1.0^,). In 4-7 again only three skill areas were- at or r 
•above 50Acoverage: computational skills (100^.), mathematical 
aopUcatIV83.3!, and problem solving skills (100^.). Generally, 
the m^jor Evaluative inadequacy of the test items appears to 
be their tfaditional nature while the mathematics objectives 
selected and developed for the study reflect the revisions inade 
lYi the field over ta,e^.past decade. 

r 

Obje'ctives developed in the area of Elementary Science 
were designed to differentiate among skills acquired m grade • 
levels 1-4 and-5-7. The organization of -the ob]ect3.vcs \ 
illustrates continuing development in twenty-four basic skill 
areas. The achievement series (1-4) arid (5-7) were validated 
by the^ objectives at ea«K level. . * ' • 

Grade level 1-4 included objectives topically organized in 
all twenty-four of the skill areas. Of. these twenty-four, only 
three"^ skill areas w6re at or above 50%. coverage : know and use 
terms, concepts and principles in each science (50%) ; ^PPly 
principles of science toward a better interpretation of their 
natural environment (100%); and apply scientific principles to . ^ 
the solution of problems in new situations (75.0%). ^ ^ • 

Objectives developed for grade level 5-7 were a continuation 
of topical Skill acqu-isition in fifteen ofo the twenty-four ; 
skill areas included in grade level 1-4. Again, ^^^^ • ,^ 

topical skill areas were covered at /r above the 50^ level by 
the achievement series: know and use' terms, concepts and 
principles in earth sciertce (66.7%;. know and use terms, concepts 
and principles in space science (50.0-0); and construct and use 
classification schemes in terns of properties involved (75. Oo). 
The tests seemed to emphasize information recall in traditional 
areas of science with little attentiori to assessment of recent 
advances in the field. 
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Implication s 

The inves'fcigators rccoanize, as do others, "dhe' usefulness 
of nationally. 3 tdndardizca tests. They nake possible intra- 
division and ihter^-ulvision normative -aciiicvement comparisons 
in the cdnnitive domai,n anii enable a state tO'Conp-are pupil 
perforfaance y!it'i\ normative ''performance of pupils in the natron. 

t It is generally recognised, tliough, that - curriculum 
disparities handicap ciie success of national standardized tests 
to measure in a state or district pi: school .specific instructional 
objectives and .outcomes in most, if not, al 1-, ' sub jcct areas. / 
Generally, aonievenent test batteries intended -for national use ^ 
endeavor to strike a compromise in terms of coverage arid grade 
placement of cognitive conterft. «- ^ ' . * / 

, .-f 'Another limitation of national ^hiev^ement tests,:, in^the 
opinion oT^^ny, is tne fact that sinqfl^e item perfcrrtance as 
a mastery measure is .lost by summation of such pcr^^ormances to 
obtain suotest or total scores for normative developmental 
interpretation . ' ^ ' ' * 

The investigators ' suggest that local school systems develop 
their'wcognitive objectives, since objectives are non-pluralistic, 
and then develop or seek helpvto develop criterion-referenced 
exercises to measure attainment of these objectives in at 
least the basic skills and desirably afll subject matter, areas. 

.Such is not an easy t.ask to accomplish nov/. It requires 
local fiscal and. perscnriel rcsourcesx* expertise and cooperation 
in 9urriculum and evaluation design. One thing seems certain: 
when the demand for .criterion-referenced tests approaches the 
present request for normative-referenced tes^ts, then agencies 
external to the local school system will increasingly prpvide 
the needed assistan^-je and prdrduct. It has already beguii^ 
However accomplished, this 'type of evaluation is essential to 
a valid diagnosis of curriculum success and failure, and n'eeded"- 
change. , ^ , . ^ 



